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Background / Context: 

There is a dearth of literature that use research design for causal inference that estimate the effect 
of ICT programs on literacy outcomes in early primary, particularly in Sub-Saharan Africa. 
There are several programs that have used ICT at a large scale, including Los Angeles, Peru, 
Nicaragua, Rwanda and an ongoing program in Turkey. Seldom have the studies directly 
estimated the effect of the ICT program on learning, as the measures used have typically been a 
middle level of the causal chain (Strigel & Pouezevara, 2012). 

Mobile phones are increasingly available in the market, and several authors argue that mobile 
learning is an increasingly ideal way to increase outcomes at scale (UNESCO, 2012; GSMA, 
2012; McKinsey & Co., 2012; Vosloo, 2012). There are concerns that using mobiles for learning 
would differentiate by gender (Murthy, 2011). Fewer studies have estimated the impact of e- 
readers on learning, though some recent studies in Kenya and Ghana are promising 
(Worldreader, 2012). 

Analyses of the impact of large scale tablet and laptop initiatives is somewhat discouraging. The 
World Bank blog of ICT Specialist Michael Trucano recently presented an analysis of several 
countries programs, including the USA, Uruguay, Thailand, Peru, Kenya, Rwanda, Turkey, 
India, Argentina and Portugal (Trucano, 2013). Few of these studies consider cost when 
estimating impact, and the impacts identified remain small or not yet determined. In all of the 
studies researched, in fact, there was no attempt to compare the cost and effectiveness of 
different ICT interventions within one study. 

Purpose / Objective / Research Question / Focus of Study: 

The Kenya PRIMR Initiative aims to improve literacy learning by aligning learning materials 
and teacher practices with current research, providing ongoing instructional support and 
observation, and supplying instructional materials and English and Kiswahili books for students. 
PRIMR is a partnership between USAID/Kenya and the Ministry of Education, with 
implementation by RTI International. Embedded in PRIMR is a study of the effectiveness and 
cost-effectiveness of ICT in Kenya, located in Kisumu county. 

The fundamental question that the Kisumu county ICT intervention study answers is how and 
whether a set of ICT interventions in Kenya improve pupil achievement in Class 2. The use of 
ICT is intended not to introduce teachers and pupils to ICT, but to determine whether and how 
ICT can improve and simplify teaching and learning. The design of the ICT program is to 
determine the effect of ICT when it is embedded within a larger instructional reform program. 
This is notable since we can compare ICT options against each other rather than simply against 
control. In this study, each ICT intervention is being compared against each other and the control 
group to measure effectiveness and cost-effectiveness. As noted, the ICT trial is being 
implemented and managed by the PRIMR Initiative in Kisumu County. 
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Setting: 


PRIMR’s ICT study is located in Kisumu county. Zones of schools within Kisumu were 
randomly assigned by zone to three treatment and one control groups. Kisumu county is in the 
western part of Kenya and borders Lake Victoria. It includes Kisumu town, one of the larger 
towns in Kenya. Zones were randomly selected from the entire list of zones in Kisumu, stratified 
by urbanicity. The zones were then assigned to treatment groups so that each treatment group has 
a randomly selected rural and peri-urban zone. 

Population / Participants / Subjects: 

The ICT-based intervention consists of three treatment groups of 20 schools, as follows: (1) The 
first group is using tablets to bolster the Teachers’ Advisory Centre (TAC) Tutor instructional 
support system. (2) In the second subgroup, teachers are using enhanced teachers’ guides on 
tablets, specially fitted with PRIMR instructional materials, as well as specially designed 
classroom pedagogical supports, including virtual flashcards, additional reading materials, and 
the RTI-developed continuous-assessment software called TangerineiClass™. This treatment is 
being implemented. (3) The third subgroup of schools is using e-readers to help pupils practice 
their reading skills. Finally, 20 schools (2 zones) are serving as controls, but will have 
intervention support in 2014. Table 1 presents the population of schools and pupils in each 
treatment group. 

Intervention / Program / Practice: 

The core PRIMR instructional intervention included the following components: Inexpensive 
books, limited instructional aids, self-contained lesson plans, supplementary readers, modest 
training and focused classroom observations. The ICT sub-study included these same 
interventions, as well as the additional ICT components below. The three ICT interventions 
differed in the level at which the ICT was applied. The three treatment groups were: 

• E-readers for pupils. Pupils in this treatment group received a basic Amazon Kindle e- 
reader pre-loaded with the English and Kiswahili PRIMR book, and more than 200 titles 
appropriate for Class 2 learners in English, Kiswahili and the local language of the area, 
called Dholuo. Each school in the e-reader intervention had an afternoon session for pupils 
which allowed them to read the supplementary books or PRIMR books at their leisure. The 
intervention was otherwise the same as the rest of the PRIMR intervention. 

• Tablets for teachers. Teachers in this treatment group each received a Google Nexus tablet. 
The tablet was pre-loaded with multimedia versions of the lesson plans that allowed the 
pupils to hear pronunciations of the words and sounds, as well as dramatic readings of the 
pupil stories and teacher read-alouds. The tablets also were pre-loaded with a virtual flash 
card program that allowed the teachers to help the pupils become comfortable with letter 
sounds in both Kiswahili and English. The tablets also had a program designed to provide 
continuous assessment feedback to the teachers based on progress with PRIMR lessons. 

• Tablets for TAC tutors. The third treatment group in the Kisumu study provided TAC 
tutors, or instructional coaches/supervisors, with a tablet. This tablet was loaded with tools to 
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help the TAG tutors support the teachers in implementing the program, similar to the paper 
tools used in other zones. 

The ICT sub-study is therefore a comparison of not only the effectiveness of ICT in comparison 
to control, but in comparison to other ICT interventions at different levels. The study includes a 
cost element as well, as the additional cost of the programs is compared with the control 
scenario, and the treatment groups are compared against each other. 

Research Design: 

All four treatment and control groups were subjected to a rigorous set of baseline (January 2013), 
midterm (June 2013) and final (October 2013) assessments, using the Early Grade Reading 
Assessment (EGRA) tools to determine what causal effect each ICT intervention has on student 
achievement. This design, mixed with the randomized assignment of schools to the interventions, 
allow us to estimate what impact each ICT intervention has on student achievement. The design 
will also allow for estimation of whether and how the ICT interventions affected student 
outcomes over and above the standard PRIMR package of interventions. 

Data Collection and Analysis: 

Data was collected by Kenyan field staff who had worked with PRIMR lead implementer RTI 
International since 2007 on several studies using EGRA. The January 2013, June 2013 and 
October 2013 assessments utilized standardized interrater reliability measures that showed .95 
reliability scores across the measures. Utilization of electronic data collection tools, specifically 
Tangerine, have removed data entry errors and standardized cleaning codes in Stata simplify the 
cleaning processes dramatically. 

Data analysis was undertaken using Stata, and the svy subset of commands was utilized to 
account for the sampling decisions. The svy commands also account for the nested nature of the 
data and groups the standard errors at the correct level. Analysis was quite simple. Given the 
randomized control design, that utilized random selection and random assignment methods, 
simple comparisons between treatment and control can be used to estimate impact. However, 
given that there were a few items where there were differences in outcomes between treatment 
and control groups, we supplement the basic analyses with a difference in difference model. 
Effect size calculations use the pooled standard deviations between the various assessments. 

Findings / Results: 

The June 2013 midterm assessment showed significant gains in English and Kiswahili letter 
sound fluency and oral reading fluency, and modest gains in reading comprehension for all 
treatment groups compared to control, as indicated in Figure 1. The impact of all three treatment 
groups was significant. 

In order to maximize the initial gains made in the PRIMR ICT zones for the remaining 
intervention period in Kisumu and also in other parts of Kenya, we submit the following 
recommendations. The ICT analysis shows statistically significant differences between all three 
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treatment groups (e-readers for students, tablets for teachers and tablets for TAG tutors) and the 
control group at the midterm. However, there were no statistically significant differences 
between the three treatment groups. Additional analyses (see Figure 2) show the cost 
effectiveness of the programs and the evidence is strong that the e-reader program is far less 
cost-effective than the other treatment conditions, including control. 

Conclusions: 

With respect to the ICT experiment, there is little evidence available on the effects of ICT 
interventions in sub-Saharan Africa. In addition, what little evidence exists suggests that ICT 
programs are not often ineffective. However, in a context where printed material is quite 
expensive, e-readers may dramatically expand access to print. Similarly, the lack of integration 
of ICT into the formal system is often decried by the Kenyan Ministry of Education. The ICT 
program is under careful examination in order to determine the lessons learned for Kenya’s own 
ICT program, with an emphasis on rapidly expanding access to ICT for primary school. 
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Appendix B: Tables and Figures 


Table 1. Treatment groups for Kisumu ICT Study 


Treatment Group 

Intervention 

Schools 

Pupils 

PRIMR -F TAC tutor 
tablet 

TAC Tutors with tablets for supervision and 
assessment 

20 

701 

339 

PRIMR -F teacher tablet 

Teachers with tablet with multimedia lesson 
plans, virtual flashcards, and continuous- 
assessment tools 

20 

432 

452 

PRIMR -F pupil e- 
reader 

Pupils with e-readers, loaded with PRIMR 
content, Kenyan textbooks, and supplementary 
reading materials 

20 

603 

328 

Control 

None 

20 

1000 


Figure 1. Gains in English and Kiswahili letter sound fluency and oral reading fluency, by 
ICT treatment 
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Figure 2. Cost-effectiveness of ICT Options in Kenya, at midterm (June 2013) 
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